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GAME-BASED  TRAINING  EFFECTIVENESS  EVALUATION  IN  AN  OPERATIONAL 
SETTING 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Army  trainers  are  increasingly  turning  to  emerging  technologies  and  innovative  training 
methods  to  exploit  low-cost,  technology-based  solutions  to  rapidly  fill  critical  training  gaps  and 
increase  the  impact  and  effectiveness  of  training  for  our  Soldiers  in  a  time  of  war.  Unfortunately, 
the  empirical  research  linking  game-based  training  to  training  effectiveness  is  limited  and  does 
not  extend  much  beyond  extolling  the  motivational  benefits  of  games  in  training.  The  Army 
lacks  empirical  evidence  on  which  to  base  future  decisions  regarding  the  acquisition  and  use  of 
game -based  training.  Therefore,  the  U.S.  Army  Research  Institute,  under  the  sponsorship  of  the 
U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Capability  Manager  -  Gaming  (TCM 
Gaming),  conducted  an  analysis  to  empirically  investigate  the  effectiveness  of  game-based 
training  in  Army  operational  settings.  In  addition,  this  analysis  also  sought  to  understand  why 
games  may  be  useful  training  tools  and  if  there  are  situational  contexts  that  dictate  the  extent  of 
their  effectiveness.  The  gaming  platfonn  Virtual  BattleSpace2  (VBS2):  U.S.  Anny  was  used  as 
a  method  for  evaluating  the  merits  of  game -based  training. 

Procedure: 

In  order  to  develop  an  appropriate  evaluation  protocol,  we  observed  training  at  Ft.  Hood, 
TX  and  Ft.  Lewis,  WA.  These  observations  provided  infonnation  about  the  flow  of  training 
using  VBS2:  U.S.  Army  and  also  provided  information  about  the  training  objectives  normally 
targeted  during  training.  The  observations  at  the  training  sites  also  highlighted  several  constraints 
that  had  to  be  taken  into  consideration  when  developing  the  research  protocol.  First,  we  could 
not  interfere  with  the  training.  Second,  the  amount  of  time  that  was  made  available  for  data 
collection  was  limited  to  approximately  one-half  hour.  Third,  we  could  not  ask  training 
facilitators  to  evaluate  the  perfonnance  of  the  unit.  Therefore,  based  on  these  constraints  and  the 
other  information  gathered  during  the  observations,  we  developed  hypotheses  and  measures. 

Data  were  collected  from  165  Soldiers  engaging  in  training  using  VBS2:  U.S.  Army  at 
Ft.  Hood  and  Ft.  Lewis.  Trainees  completed  pre-  and  post-training  measures  during  the  course  of 
their  nonnal  training  experience.  Measures  of  learning  at  both  the  individual  level  (e.g., 
individual  perfonnance  on  the  tasks)  and  the  unit  level  (e.g.,  working  together)  were  included  as 
measures  of  effectiveness.  In  addition,  due  to  the  variance  in  training  both  within  and  across 
training  locations,  measures  to  understand  the  context  under  which  training  may  be  effective 
were  also  administered.  Contextual  variables  considered  included  the  difficulty  of  the  training 
mission,  the  level  of  unit  preparation  prior  to  training,  the  level  of  leader  involvement  during  the 
training,  and  time  spent  on  the  training. 
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Findings: 


Results  of  the  evaluation  demonstrate  that  the  training  had  a  positive  impact  on  individual 
outcomes,  as  well  as  on  unit  learning.  Specifically,  Soldiers  reported  being  more  prepared  to  plan 
tactical  convoy  operations  following  the  training.  In  addition,  higher  levels  of  Unit  Process  and 
Unit  Cohesion  were  reported  following  the  training.  These  unit-level  findings  illustrate  that  the 
training  impacted  how  well  the  Soldiers  felt  they  could  work  together  as  a  team,  as  well  as  their 
attraction  to  the  unit,  including  their  attraction  to  their  tasks  and  other  group  members.  In 
addition,  contextual  variables  were  shown  to  influence  both  individual  and  unit  level  outcomes. 
Both  the  level  of  unit  preparation  for  the  training  and  the  level  of  unit  leadership  involvement 
during  exercise  management  positively  affected  training  outcomes.  Together  these  results 
illustrate  that  game -based  training  (GBT)  can  be  an  effective  training  tool  to  the  degree  that  the 
training  has  specific  objectives  which  are  focused  on  during  the  training.  This  focus  can  come 
from  preparing  for  the  training  ahead  of  time  and  from  leaders  being  involved  during  the  training 
to  provide  focus  and  feedback. 

Utilization  and  Dissemination  of  Findings: 

A  draft  report  of  the  results  was  provided  to  TMC  Gaming  19  March  2010.  Results  were 
also  presented  at  the  Defense  GameTech  Conference  3 1  March  2010.  They  will  be  used  to 
support  future  decisions  regarding 
the  acquisition  and  use  of  GBT  in  the  U.S.  Army. 
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GAME-BASED  TRAINING  EFFECTIVENESS  EVALUATION  IN  AN  OPERATIONAL 

SETTING 

Introduction 

The  U.S.  Army  faces  significant  training  challenges,  now  and  in  the  years  ahead.  Some  of 
these  challenges  involve  persistent  conflict  and  deployment  pressures  on  training  requirements 
and  time.  An  increased  diversity  of  operational  environments,  the  complexity  of  possible 
domestic  and  global  missions,  and  competing  requirements  for  training  resources  are  all  key 
factors  that  have  a  potential  impact  on  future  training  budgets  (McHugh  &  Casey,  Jr.,  2010).  The 
Army  needs  methods  for  providing  Soldiers  and  leaders  with  effective  training  and  opportunities 
to  practice  tasks  effectively  and  efficiently.  Consequently,  Anny  trainers  are  increasingly  turning 
to  emerging  technologies  and  innovative  training  methods  to  exploit  low-cost,  technology-based 
solutions  to  rapidly  fill  critical  training  gaps  and  increase  the  impact  and  effectiveness  of  training 
for  our  Soldiers  in  a  time  of  war. 

One  technological  approach  to  training  that  has  received  increased  interest  over  the  last 
decade  is  the  use  of  game-based  training  (GBT).  Training  researchers  hypothesize  that  games  are 
instructive  because  they  engage  the  player/leamer,  leading  to  skill  acquisition  and  retention 
(Colquitt,  LePine,  &  Noe,  2000;  Mathieu,  Tannenbaum,  &  Salas,  1992;  Prensky,  2001; 
Tannenbaum  &  Yukl,  1992).  In  fact,  empirical  research  has  provided  evidence  that  attributes  of 
games  (e.g.,  challenge,  realism,  and  interactivity)  influence  trainee  motivation  and  the  length  of 
time  that  trainees  are  willing  to  invest  in  mastering  the  skills  taught  during  game  play  (e.g., 
Corbeil,  1999;  Engel,  et  al.,  2009;  Garris,  Ahlers,  &  Driskell,  2002). 

Many  Anny  agencies  and  organizations,  including  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC),  have  recognized  that  games  have  the  potential  to  augment  and  improve 
military  training  for  both  individuals  and  collectives.  Examples  of  active  GBT  applications 
include  Army  DARWARS  Ambush!  and  Tactical  Iraqi.  These  games  are  being  used  to  train 
Soldiers  on  a  variety  of  skills  including  convoy  operations,  language,  cultural  awareness,  and 
other  tactical  drills  and  tasks. 

The  most  recent  GBT  effort  is  the  acquisition  and  fielding  of  Virtual  BattleSpace2 
(VBS2):  U.S.  Army.  VBS2:  U.S.Army  is  based  on  the  commercial  gaming  engine  Real 
Virtuality  2  created  by  Bohemia  Interactive.  It  is  an  interactive,  three-dimensional  training 
system  to  provide  a  synthetic  environment  on  networked  PCs  suitable  for  a  wide  range  of 
military  training  and  experimentation  purposes.  Trainees  typically  interact  with  the  simulation 
using  a  keyboard  and  mouse,  although  simulated  vehicle  control  interfaces  (steering  wheel  and 
pedals)  can  be  used.  It  offers  large,  realistic  terrain  areas  (up  to  100km  x  100km  in  size).  VBS2: 
U.S.  Army  includes  after-action  review  capability,  High  Level  Architecture/Distributed 
Interactive  Simulation  (HLA/DIS)  compliance,  and  a  mission  editor  that  allows  scenarios  to  be 
created  and  also  modified  in  real  time.  VBS2:  U.S.  Army  places  military  trainees  in  realistic 
urban  or  rural  settings,  and  provides  them  with  the  opportunity  to  practice  protocols  relating  to 
convoy  and  ambush  operations.  Trainees  can  drive  Army  vehicles,  pilot  aircraft,  and  fire 
weapons  across  a  number  of  game  scenarios.  The  use  of  VBS2:  U.S.  Army  is  designed  to  be  a 
stepping  stone  in  Soldier  training.  Ideally,  units  will  use  the  game  to  practice  tactics,  techniques 
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and  procedures  prior  to  going  into  the  field.  By  engaging  in  GBT,  Soldiers  can  improve  their 
skills  prior  to  participating  in  a  live  training  exercise. 

A  VBS2:  U.S.  Army  screenshot  is  shown  in  Figure  1.  Further  information  about  VBS2:  U.S. 
Army  can  be  found  at  the  Anny  MILGaming  Portal  (https://milgaming.army.mil). 


Figure  1.  A  VBS2:  U.S.  Army  screenshot. 

While  the  Anny  has  established  a  game  technology  program  of  record,  there  are  still 
many  unanswered  questions  regarding  game  technology  training  effectiveness.  Much  of  the 
common  wisdom  concerning  the  effects  of  GBT  is  anecdotal,  lacking  an  empirical  foundation 
(Hays,  2005).  The  majority  of  articles  and  reports  written  and  published  on  instructional  games 
are  based  on  the  writers’  opinions  of  the  promise  of  instructional  games  (Rieber,  Smith,  &  Noah, 
1998).  Few  articles  have  empirically  studied  the  effectiveness  of  instructional  games  for  either 
individual  or  team  training  (Hays,  2005).  The  research  that  does  exist  has  been  focused  on 
enhancing  our  understanding  of  how  to  design  training  games  to  keep  trainees  interested,  with 
less  emphasis  placed  on  how  games  influence  the  acquisition  of  specific  knowledge  and  skills. 
Just  because  a  game  is  motivating  to  play  does  not  mean  that  it  will  be  instructional.  In  fact,  what 
is  motivating  about  a  game  may  hinder  learning.  Research  has  shown  that  if  the  content  is  not 
correctly  embedded  into  the  game,  learners  are  more  likely  to  walk  away  with  increased 
knowledge  of  how  to  play  the  game  rather  than  the  transferable,  operational  knowledge  or  skill 
(Belanich,  Sibley,  &  Orvis,  2004). 

The  existing  body  of  research  does  not  provide  insight  as  to  how  well  GBT  works 
relative  to  other  methods,  nor  why  or  when  (under  what  conditions)  to  use  it.  Decisions  to  use 
games  to  train  have  been  based  on  a  “leap  of  faith,”  and  there  have  been  few  efforts  to  test  that 
faith  through  rigorous  evaluation  of  the  hypothesized  benefits  of  games  over  other  training 
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methods  (Hays,  2005).  Training  designers  need  to  understand  how  to  incorporate  game-based 
approaches  to  provide  training  that  is  pedagogically  sound,  as  well  as  motivating  and  engaging. 
The  credibility  and  impact  of  GBT  programs  could  rise  dramatically  if  solid  empirical  research 
was  conducted  on  the  effectiveness  of  such  training  approaches. 

During  FY  2009,  the  Anny  distributed  copies  of  VBS2:  U.S.  Army  to  many  training 
locations  all  over  the  world.  With  such  an  aggressive  application  of  GBT,  there  is  a  need  to 
evaluate  the  effectiveness  of  the  platform  in  both  operational  and  institutional  settings.  In 
response  to  this  need,  the  U.S  Army  Research  Institute  for  Behavioral  and  Social  Sciences 
(ARI),  under  the  sponsorship  of  TRADOC  Capability  Manager  Gaming  (TCM  Gaming), 
conducted  a  two-part  effort  to  assess  the  effectiveness  of  the  use  of  computer  games  for  Anny 
training.  Both  efforts  use  VBS2:  U.S.  Army  as  the  evaluation  platfonn.  The  work  described  in 
this  report  focuses  on  evaluating  the  use  of  VBS2:  U.S.  Army  in  the  operational  (unit)  setting, 
while  the  other  effort  focuses  on  the  use  of  GBT  in  the  institutional  setting. 

The  research  literature  indicates  that  training  effectiveness  is  influenced  by  three  primary 
determinants:  a)  the  training  program  itself  (including  the  game),  b)  the  trainee  (in  terms  of 
personal  characteristics  and  attitudes),  and  c)  the  situational  context  in  which  the  training  takes 
place  (Campbell  &  Kuncel,  2001;  Colquitt,  et  al.,  2000;  Mathieu,  et  ah,  1992;  Salas  &  Cannon- 
Bowers,  2001).  In  addition  to  studying  the  overall  effectiveness  of  VBS2:  U.S.  Army,  the 
purpose  of  this  analysis  was  to  investigate  the  impact  of  situational  variables  on  training 
outcomes.  Such  a  multi-pronged  approach  not  only  evaluates  the  overall  effectiveness  of  the 
GBT  but  also  explains  “why”  the  game-based  approach  may  or  may  not  be  effective,  and 
therefore,  provides  suggestions  for  improving  future  effectiveness  of  GBT. 

This  report  first  describes  the  method  associated  with  developing  the  evaluation  protocol. 
This  method  consisted  of  making  site  visits  to  Army  posts  using  VBS2:  U.S.  Army  on  a  regular 
basis.  Next,  the  evaluation  protocol  is  described,  along  with  the  associated  measures  and 
hypotheses.  Finally,  this  report  presents  the  evaluation  results  and  also  makes  recommendations 
for  the  use  of  GBT  in  an  operational  setting. 
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Method 


Site  Observations 

Prior  to  developing  the  method  by  which  evaluation  data  would  be  collected,  we 
conducted  site  visits  to  locations  using  VBS2:U.S.  Army.  The  purpose  of  those  visits  was  to 
leam  how  training  was  conducted  at  each  site  in  order  to  develop  an  appropriate  evaluation  plan 
that  matched  the  training  method  used. 

On  11-13  August  2009  and  18-19  August  2009,  site  visits  were  conducted  at  Ft.  Hood, 
TX  and  Ft.  Lewis,  WA,  respectively.  These  sites  were  chosen  because  both  were  using  VBS2: 
U.S.  Army  for  training  on  a  regular  basis.  Throughout  the  five  days  of  observation,  a  number  of 
units  came  into  the  training  centers  to  train  using  VBS2:  U.S.  Army.  Although  the  primary 
training  objective  across  the  two  sites  was  convoy  training,  VBS2:  U.S.  Anny  also  supported 
additional  training  objectives  including  crew  coordination  and  teamwork;  communications;  and 
reinforcing  rules  of  engagement  (ROEs)  or  standard  operating  procedures  (SOPs).  Although 
there  were  some  differences  across  the  two  sites  in  tenns  of  the  training  logistics,  the  overall 
flow  of  the  training  was  the  same.  The  next  section  of  this  report  describes  that  training. 

Training  using  VBS2:  U.S.  Army.  At  both  Ft.  Lewis  and  Ft.  Hood,  each  training  facility 
contained  computers  for  each  individual  Soldier  to  use  during  the  training  session.  After  the 
training  unit  entered  the  training  facility,  each  Soldier  was  assigned  a  different  role  or  position 
for  the  convoy  scenario  -  driver,  gunner,  truck  commander,  or  observer.  One  Soldier  was  also 
assigned  the  position  of  convoy  commander.  In  addition,  the  role  of  higher  headquarters  is  filled 
by  either  a  training  facilitator  or  someone  in  a  leadership  position  (e.g.,  company  commander) 
within  the  unit.  By  having  all  of  these  roles  filled  during  the  training,  the  Soldiers  are  able  to  get 
a  sense  of  what  each  role  does,  learn  how  to  utilize  each  position,  and  also  learn  to  convey  the 
appropriate  information  to  the  appropriate  person  at  the  appropriate  time. 

At  each  training  site,  there  was  the  potential  for  wide  variability  in  the  computer  skills 
and  experience  with  games  across  all  of  the  Soldiers.  Therefore,  before  beginning  actual 
missions,  Soldiers  were  all  given  initial  training  on  how  to  use  the  game  in  order  to  get  all 
trainees  comfortable  with  the  controls.  Following  this  initial  training  period,  the  training  unit  was 
given  its  mission,  and  planning  for  the  mission  commenced.  All  Soldiers  were  then  briefed  on 
the  mission  (typically  by  the  convoy  commander)  and  given  time  to  ask  questions.  Following  the 
mission  brief,  Soldiers  then  went  through  the  mission.  Throughout  the  mission  scenario,  training 
facilitators  played  the  role  of  the  opposing  forces  (OPFOR)  and  embedded  attacks  in  the  form  of 
insurgents,  snipers,  and  improvised  explosive  devices  (IEDs)  into  the  scenario.  The  unit  had  to 
achieve  their  mission  objectives  while  contending  with  attacks  from  the  OPFOR.  During  the 
observed  training,  it  was  not  uncommon  for  a  Soldier's  avatar  to  experience  a  casualty  during  the 
scenario.  All  of  these  elements  added  a  sense  of  stress  to  the  training  mission,  encouraging  the 
Soldiers  to  take  the  training  seriously. 

Finally,  after  the  mission  was  complete,  a  thorough  after-action  review  (AAR)  was 
conducted.  After  completion  of  the  AAR,  units  could  conduct  a  second  mission,  following  the 
same  procedure  as  described  above.  Depending  upon  the  unit  commander’s  plan,  the  second 
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mission  could  be  identical  to  the  first  one  conducted  or  could  be  different  and  possibly  more 
difficult. 


Across  the  observed  missions,  there  was  a  lot  of  variance  in  who  led  and  conducted  the 
AAR.  For  some  units,  the  AAR  was  led  by  a  higher-level  leader  who  had  not  actually 
participated  in  the  training.  For  other  units,  it  was  led  by  the  individual  given  the  position  of 
convoy  commander  during  the  training,  and  hence,  was  deemed  “the  leader”  of  the  exercise.  In 
other  instances,  the  AAR  was  conducted  by  a  Soldier  who  had  been  selected  to  conduct  the  AAR 
at  the  completion  of  the  exercise.  Finally,  for  some  units,  the  AAR  was  led  by  one  of  the  training 
facilitators  and  not  by  a  Soldier  or  higher-level  leader  in  the  training  unit.  This  variance 
contributed  to  some  AARs  being  conducted  more  thoroughly  than  others. 

One  of  the  unique  features  of  VBS2:  U.S.  Army  is  that  it  has  the  capability  to  play  back 
the  entire  mission,  including  from  a  birds-eye  view,  for  use  during  the  AAR.  This  capability 
allows  Soldiers  to  see  things  like  where  the  insurgents  were  shooting  from,  what  their  truck 
formation  looked  like,  and  how  each  individual  Soldier  was  reacting  and  behaving.  Getting  this 
overall  view  of  the  mission  can  provide  Soldiers  with  a  better  and  more  objective  understanding 
of  their  behaviors  and  the  results  of  those  behaviors.  Although  the  playback  capability  is 
available,  there  were  differences  in  how  well  the  recording  was  integrated  into  the  AAR.  At 
times,  the  video  playback  capability  was  not  utilized,  or  only  partially  used. 

These  observations  provided  insight  into  the  types  of  measures  that  should  be  used  in  the 
evaluation.  These  visits  also  highlighted  the  challenges  associated  with  conducting  the 
evaluation.  One  of  the  things  that  stood  out  was  the  fact  that  there  were  differences  not  only 
between  Ft.  Hood  and  Ft.  Lewis,  but  also  within  each  site  in  terms  of  how  the  training  was 
conducted.  For  example,  some  units  spent  a  much  longer  time  planning  their  missions  than 
others.  Some  units  had  the  involvement  of  very  senior  officers,  such  as  lieutenant  colonels, 
whereas  other  units  had  no  leadership  present  other  than  those  participating  in  the  training 
scenarios,  and  others  had  some  level  of  leadership  present  only  observing,  not  participating. 
Another  source  of  variability  observed  was  the  training  scenarios  used.  Because  each  unit  came 
to  the  training  with  its  own  unique  training  needs  and  objectives,  it  was  nearly  impossible  to 
develop  objective  measures  of  perfonnance  prior  to  conducting  the  evaluation.  It  became  clear 
that  the  only  way  to  control  for  all  of  those  factors  was  to  measure  them  to  the  greatest  extent 
possible  while  keeping  in  mind  that  the  actual  data  collection  time  was  limited.  In  short,  the 
evaluation  plan  had  to  be  designed  to  try  and  overcome  some  of  these  challenges  associated  with 
the  training. 

Data  Collection  Protocol  and  Hypothesis  Development 

Following  training  observations  at  Ft.  Lewis  and  Ft.  Hood,  the  data  collection  protocol 
was  developed.  Although  time  to  administer  measures  was  limited,  we  were  given  pennission 
from  each  training  facility  to  administer  a  short  set  of  measures  both  prior  to  and  after  training 
was  complete.  Therefore,  measures  to  administer  both  pre-  and  post-training  were  developed. 

The  hypotheses  and  measures  generated  as  part  of  the  protocol  focused  on  measuring  the 
skills  that  VBS2:  U.S.  Anny  targeted  during  training.  Training  seemed  to  focus  on  the 
development  of  two  areas:  skills  associated  with  individual  level  learning  and  skills  related  to 
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learning  at  the  unit  level.  Therefore,  the  measures  generated  for  this  evaluation  also  focused  on 
measuring  these  two  different  types  of  outcomes.  In  addition,  the  data  collection  protocol 
included  measures  of  the  varying  contextual  differences  observed  across  the  training  sessions. 
The  majority  of  measures  were  administered  both  pre-  and  post-training.  The  difference  between 
the  two  administrations  was  in  the  question  stems.  The  post- training  items  asked  the  trainees  to 
respond  to  the  items  now  that  they  have  engaged  in  training  using  VBS2:  U.S.  Army. 

Individual  level  measures.  Given  the  variation  in  the  technical  training  objectives  across 
training  scenarios,  it  was  difficult  to  develop  measures  to  assess  individual  level  learning. 
However,  based  on  our  observations  and  conversations  with  the  training  facilitators,  we 
developed  measures  associated  with  specific  behaviors  and  tasks  that  are  typically  performed 
while  engaging  in  training  using  VBS2:  U.S.  Army.  For  example,  specific  behaviors  and  tasks 
associated  with  conducting  convoy  operations  (e.g.,  scanning  sectors  of  fire,  communicating 
with  unit  members,  locating  sources  of  enemy  fire,  and  identifying  civilians)  were  identified  and 
used  in  measures  associated  with  performing  these  skills  during  training. 

The  individual  level  measures  administered  consisted  of  a  demographics  measure  (8 
items;  Appendix  A,  pg.  23),  skill  preparedness  measure  (13  items;  Appendix  A,  pg.  24),  training 
motivation  (10  items;  Appendix  A,  pg.  25)  and  task  perfonnance  (15  items;  Appendix  A,  pg.  26). 
For  the  majority  of  these  measures,  trainees  responded  on  a  5-point  scale  ( 1  =  Strongly  Disagree; 
5  =  Strongly  Agree).  When  necessary,  items  were  re-coded  so  that  higher  numbers  represented  a 
more  positive  score.  Table  1  lists  when  each  measure  was  administered  (e.g.,  pre-  or  post¬ 
training),  a  description  of  the  measure,  and  an  example  item  for  each  measure. 

Unit  level  measures.  Several  measures  were  used  to  assess  unit  level  outcomes. 
Examining  these  measures  over  time  provides  evidence  of  the  training  impact  on  unit  outcomes. 
These  unit  level  variables  included  the  following:  unit  process  measure  (13  items;  Appendix  A, 
pg.  27),  unit  cohesion  measure  (task  and  interpersonal  cohesion  for  a  total  of  1 1  items;  Appendix 
A,  pg.  28),  unit  efficacy  measure  (7  items;  Appendix  A,  pg.  29),  and  overall  unit  effectiveness 
measure  (3  items;  Appendix  A,  pg.  29).  All  of  these  measures  were  administered  both  pre-  and 
post-training.  More  infonnation  about  each  of  these  measures  is  in  Table  1. 

Contextual  variables.  In  order  to  understand  the  context  under  which  GBT  may  be  more 
or  less  effective,  we  also  generated  several  measures  to  assess  variables  that  may  impact  the 
effectiveness  of  the  training.  These  measures  were  completed  by  outside  observers  to  the 
training.  All  measures  assessing  contextual  variables  are  in  Appendix  A. 
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Table  1.  Information  about  Individual  and  Unit  Level  Measures 


Measure 

Measure  Description 

When 

Administered 

Example  Item(s) 

Skill  Preparedness 

Assess  Soldier  belief  in  level  of 
preparation  to  perform  certain 
general  tactical  tasks  often  a  part 
of  VBS2  training  missions 

Pre-  and  Post 
Training 

How  prepared  do  you  feel 

to... 

“Assess  the  tactical 
situation?” 

“Plan  a  tactical  convoy 
operation?” 

Training 

Motivation 

Assess  Soldier  motivation  level  to 
participate  in  VBS2  training 

Post-Training 

“I  tried  very  hard  during 
this  exercise.” 

Task  Performance 

Assess  Soldier  belief  about 
performance  on  specific  tasks 
performed  during  training 

Post-Training 

Please  rate  your  ability  to 
perform  each  of  the 
following  tasks  in  the 
simulator... 

“Scan  for  threats.” 

“Identify  insurgents.” 

Unit  Process 

Assess  Soldier  belief  about  how 
well  the  unit  worked  together 
during  the  training  in  terms  of 
different  teamwork  skills  (e.g., 
communication,  monitoring 
progress  towards  goals).  Research 
has  hypothesized  that  engagement 
in  effective  unit  processes  are 
directly  linked  to  increased  unit 
performance  (e.g.,  Marks, 

Mathieu,  &  Zaccaro,  2001). 

Pre-  and  Post 
Training 

“My  training  unit  members 
and  I  understand  how  one 
another  prefer  to 
communicate  information 
to  other  members.” 

“My  training  unit  members 
and  I  understand  how 
members  are  going  to 
work  together  to  achieve 
our  goals.” 

Unit  Cohesion 

Assess  Soldier  attraction  to  the 
unit,  including  their  tasks  (task 
cohesion)  and  other  group 
members  (inteipersonal  cohesion). 
Research  has  shown  that  unit 
cohesion  influences  unit 
performance  (e.g.  Gully,  Devine, 

&  Whitney,  1995). 

Pre-  and  Post 
Training 

Task  Cohesion'.  “When 
one  person  is  struggling 
with  a  task,  another 
member  of  the  training 
unit  will  step  in  to  help.” 
Interpersonal  Cohesion : 
“The  members  of  my 
training  unit  get  along  with 
one  another.” 

Unit  Efficacy 

Assess  Soldier  confidence  level  in 
ability  of  the  unit  to  work  together 
and  perform  well  during  the 
training.  Research  has  shown  a 
positive  relationship  between  unit 
efficacy  and  performance  (e.g. 
Stajkovic,  Lee,  &  Nyberg,  2009). 

Pre-  and  Post 
Training 

“I  am  confident  that  the 
members  of  my  training 
unit  and  I  can  effectively 
set  contingency  plans.” 

Unit  Effectiveness 

Assess  Soldier  belief  about  how 
effective  the  unit  was  during  the 
training  mission 

Pre-  and  Post 
Training 

“This  training  unit  is 
effective.” 
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Training  facilitator  measures.  The  training  facilitators  completed  measures  regarding  the 
level  of  unit  preparation  of  the  unit  prior  to  training  (Appendix  A,  pg.  30).  Facilitators  completed 
these  ratings  at  the  end  of  each  mission  completed  by  one  training  unit.  This  measure  was 
designed  to  account  for  differences  in  how  much  effort  the  unit  put  into  preparing  for  the  VBS2: 
U.S.  Army  training  prior  to  coming  to  the  training  facility.  Differences  may  exist  in  tenns  of  how 
much  the  unit  communicates  with  the  training  facilitators  ahead  of  time  in  regard  to  mission 
objectives,  training  scenarios,  etc.  For  this  measure,  facilitators  rated  each  unit  on  a  5-point  scale 
( 1  =  Completely  Unprepared;  5  =  Completely  Prepared).  These  ratings  were  averaged  across 
facilitators  if  more  than  one  facilitator  rated  the  level  of  preparation.  In  order  to  classify  the  units 
into  more  meaningful  groups,  a  dichotomous  variable  that  classified  those  units  who  received  an 
average  rating  of  3  or  less  as  “Less  Prepared”  and  those  units  who  received  an  average  rating  of 
over  3  as  “More  Prepared.”  This  dichotomy  was  used  throughout  all  analyses  involving  this 
variable.  Given  that  this  measure  was  meant  to  tap  into  the  unit’s  initial  level  of  preparation,  only 
the  rating  given  by  facilitators  regarding  the  first  mission  completed  at  the  training  center  was 
used. 


Observer  measures.  Training  observers  from  ARI,  Aptima,  and  TCM  Gaming  completed 
several  measures  as  they  observed  each  mission.  Namely,  observers  completed  Time  Spent  on 
Training  and  Leader  Involvement  measures  (Appendix  A,  pgs.  31).  For  the  Time  Spent  on 
Training  measure,  observers  simply  recorded  the  number  of  minutes  each  training  unit  spent  on 
1)  initial  VBS2:  U.S.  Army  training  (e.g.,  buttonology),  2)  the  mission  brief,  and  3)  the  actual 
training  mission.  The  time  spent  on  the  initial  training  was  only  taken  into  account  once,  whereas 
the  time  spent  on  the  mission  brief  and  the  actual  training  mission  was  taken  into  account  each 
time  the  unit  began  a  new  mission. 

The  Leader  Involvement  measure  consisted  of  six  questions  targeted  at  recording  the 
level  of  involvement  from  unit  leadership  (not  actually  involved  as  a  participant  in  the  training) 
during  each  training  mission.  The  qualitative  information  from  each  of  these  measures  was 
transformed  to  a  quantitative  three-point  scale  to  measure  the  level  of  involvement.  A  rating  of  1 
meant  that  unit  leadership  was  not  present  at  the  training  exercise;  a  rating  of  2  meant  that  unit 
leadership  participated  in  either  the  mission  brief  or  the  AAR  portion  of  training;  and  a  rating  of 
3  meant  that  unit  leadership  was  present  and  participating  in  both  of  those  important  aspects  of 
the  training. 

Hypotheses.  Based  on  all  these  measures,  we  generated  several  hypotheses  about  how 
training  would  impact  individual  and  unit  level  learning,  as  well  as  how  the  contextual  variables 
would  influence  training  effectiveness. 

1 .  VBS2:  U.S.  Army  training  positively  impacts  the  training  outcome  of  Skill  Preparedness. 

2.  The  level  of  Unit  Preparation  for  the  training  positively  impacts  the  training  outcomes  of 
a)  Skill  Preparedness,  b)  Task  Perfonnance,  and  c)  Training  Motivation. 

3.  Leader  Involvement  in  training  positively  impacts  the  training  outcomes  of  a)  Skill 
Preparedness,  b)  Task  Performance,  and  c)  Training  Motivation. 


4.  VBS2:  U.S.  Army  training  positively  impacts  the  training  outcomes  of  a)  Unit  Process,  b) 
Unit  Cohesion,  c)  Unit  Efficacy,  and  d)  Unit  Effectiveness. 

5.  The  level  of  Unit  Preparation  for  the  training  positively  impacts  the  training  outcomes  of 
a)  Unit  Process,  b)  Unit  Cohesion,  d)  Unit  Efficacy,  and  d)  Unit  Effectiveness. 

6.  Leader  Involvement  in  training  positively  impacts  the  training  outcomes  of  a)  Unit 
Process,  b)  Unit  Cohesion,  d)  Unit  Efficacy,  and  d)  Unit  Effectiveness. 

Data  Collection 

Participants.  One-hundred  and  sixty  five  Soldiers  participated  in  the  evaluation  study  76 
at  Ft.  Hood  and  89  at  Ft.  Lewis.  These  165  Soldiers  represented  9  different  platoons.  95%  of 
these  Soldiers  were  enlisted;  4%  were  officers;  and  less  than  1%  were  warrant  officers.  The 
mean  age  of  the  trainees  was  25.20  (SD  =  5.68).  In  addition,  33%  of  the  trainees  reported  having 
engaged  in  training  using  VBS2:  U.S.  Army  in  the  past. 

Procedure.  Soldiers  were  given  a  brief  overview  of  the  project  upon  entering  the  training 
facility.  They  were  informed  that  their  participation  was  voluntary  and  that  all  of  their  responses 
were  anonymous  and  would  not  be  linked  back  to  any  single  individual.  Soldiers  read  and  signed 
the  informed  consent  form,  and  then  were  given  the  pre-training  questionnaire  packet  to 
complete.  This  set  of  measures  took  approximately  15  to  20  minutes  for  each  Soldier  in  the  unit 
to  complete.  Following  completion  of  these  measures,  the  unit  engaged  in  training  as  usual  using 
VBS2:  U.S.  Army.  As  described  above,  this  training  consisted  of  a  practice  mission,  to  be 
followed  by  the  actual  training  mission,  and  an  AAR.  Soldiers  completed  a  second  set  of 
measures  after  the  final  AAR  of  the  day. 

In  addition  to  the  Soldiers  completing  measures,  after  each  mission,  training  facilitators 
provided  an  overall  rating  of  how  prepared  the  unit  was  for  the  training.  Aptima  and  ARI 
personnel  present  during  the  data  collection  also  recorded  the  time  each  unit  spent  on  training 
and  conducting  the  mission.  All  of  these  measures  served  to  control  for  some  of  the  variability 
present  across  the  training  events. 


Results 

Before  conducting  any  analyses,  the  measures  themselves  were  examined  to  ensure  that 
they  were  reliable.  Statistically,  all  of  the  items  in  each  of  the  measures  fit  well  together,  as 
demonstrated  by  a  measure  of  internal  consistency  (alpha  levels  ranging  from  .76  to  .96).  This 
statistic  indicates  that  the  items  within  one  measure  all  tap  into  the  same  construct. 

Trainees  spent  an  average  of  86  minutes  (SD  =  39.65)  on  the  initial  VBS2:  U.S.  Army 
training;  this  time  was  spent  teaching  trainees  how  to  use  the  simulation  prior  to  conducting  their 
training  mission.  The  average  time  spent  conducting  the  mission  brief  and  planning  for  the 
mission  was  17.88  minutes  (SD  =  8.96),  and  the  average  training  mission  lasted  83.64  minutes 
(SD  =  61.18). 
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Overall,  the  results  demonstrated  that  training  with  VBS2:  U.S.  Army  had  some  impact 
on  both  individual-level  and  unit-level  outcomes.  The  context  in  which  this  training  occurred 
also  impacted  both  sets  of  variables.  This  results  section  first  discusses  the  analyses  associated 
with  the  individual  level  variables,  followed  by  a  discussion  of  the  results  with  the  unit  level 
variables. 

Individual  Level  Analyses 

Hypotheses  1  through  3  focused  on  the  influence  of  the  training  and  the  training  context 
on  the  individual  level  variables  (Skill  Preparedness,  Task  Performance,  and  Training 
Motivation).  First,  in  order  to  generally  understand  how  trainee  scores  changed  from  pre-  to  post¬ 
training  measures  (and  test  hypothesis  1),  a  paired  sample  /-test  was  conducted  on  the  Skill 
Preparedness  measure  (since  pre -measures  were  not  administered  for  Task  Performance  or 
Training  Motivation,  no  analyses  could  be  conducted  on  these  measures).  Although  the  trainees 
reported  feeling  slightly  more  prepared  to  conduct  the  skills  after  training  (M=  3.97,  SD  =  .76) 
compared  to  pre-training  (M=  3.93,  SD  =  .67),  this  difference  was  not  statistically  significant, 
t(141)  =  -.61,  n.  s.  However  given  that  the  skills  each  unit  performed  during  the  training  mission 
were  aligned  with  the  diverse  training  objectives  of  each  unit,  we  looked  more  deeply  at  the 
individual  items  in  this  measure.  Upon  examination,  we  found  that  trainees  reported  feeling  more 
prepared  to  “plan  a  tactical  convoy  operation”  after  the  training  (M=  3.78,  SD  =  1.06)  than  prior 
to  the  training  (M=  3.50,  SD  =  1.15),  t  (141)  =  -2.61,/?  =  .01.  Figure  2  depicts  this  relationship. 
These  results  demonstrated  that  although  trainees  did  not  report  feeling  more  prepared  to 
perform  a  variety  of  skills  following  the  training,  they  did  report  that  the  training  made  them  feel 
more  prepared  to  engage  in  one  of  the  more  general  objectives  of  VBS2:  U.S.  Army  -  planning 
for  a  convoy  mission.  The  results  of  the  analyses  for  each  individual  Skill  Preparedness  item  are 
in  Appendix  B. 


Pre-Training  Post-Training 


*  p  =  .01 

Figure  2.  Preparedness  levels  for  planning  a  convoy  operation. 


Hypotheses  2  and  3  focused  on  the  impact  of  situational  factors  on  training  effectiveness. 
To  examine  the  influence  of  Unit  Preparation  and  Leader  Involvement  on  the  individual  level 
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outcomes,  Analysis  of  Covariance  (ANCOVA)  was  conducted.  This  statistical  analysis  technique 
was  utilized  multiple  times  while  analyzing  the  data.  ANCOVA  statistically  controls  for 
variables  that  may  mask  the  direct  relationship  between  two  other  variables.  In  all  instances 
where  ANCOVA  was  used,  the  impact  of  the  platoon  that  the  trainees  were  in,  as  well  as  pre¬ 
training  levels  of  the  variables  where  relevant,  were  controlled  for.  For  example,  hypothesis  2a 
looks  at  the  impact  of  Unit  Preparation  on  Skill  Preparedness.  In  order  to  understand  the  more 
direct  relationship  between  these  two  variables,  we  accounted  for  random  variance  associated 
with  the  training  unit  to  which  the  trainees  belonged  and  their  initial  preparedness  levels.  This 
type  of  analysis  allows  for  a  cleaner  examination  of  the  situational  variables  on  the  training 
outcome.  Therefore,  the  results  reported  here  for  Hypotheses  2  and  3,  as  well  as  for  other 
applicable  hypotheses,  are  based  on  accounting  for  those  extraneous  variables. 

For  Hypotheses  2,  ANCOVA  results  showed  that  Unit  Preparation  significantly  impacted 
a)  Skill  Preparedness,  b)  Task  Performance,  and  c)  Training  Motivation.  Figures  3,  4,  and  5 
depict  the  nature  of  these  relationships.  As  shown  in  these  figures,  trainees  in  units  which  had 
prepared  more  for  the  training  reported  being  more  prepared  to  conduct  certain  skills  and 
perform  various  tasks  throughout  the  training;  they  also  reported  higher  motivation  for  the 
training. 


'4718^ 


Less  Prepared  More  Prepared 

Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  126)  =  5.48,  rf  =  .04,  p  <  .05 
Figure  3.  Impact  of  unit  preparation  on  skill  preparedness. 
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4.5 


^3W 


Less  Prepared  More  Prepared 


Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  124)  =  7.88,  r)2  =  .06,/?  <  .01 
Figure  4.  Impact  of  unit  preparation  on  task  perfonnance. 


Less  Prepared  More  Prepared 

Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  126)  =  5.73,  rf  =  .04  ,p<  .05 


Figure  5.  Impact  of  unit  preparation  on  training  motivation. 


Hypothesis  3  was  similar  to  Hypothesis  2  and  predicted  that  the  level  of  Leader 
Involvement  in  the  training  positively  influences  a)  Skill  Preparedness,  b)  Task  Performance,  and 
c)  Training  Motivation.  Parts  b  and  c  of  this  hypothesis  were  supported;  an  ANCOVA  indicated 
that  there  was  no  statistically  significant  relationship  between  the  level  of  Leader  Involvement  in 
the  training  and  Skill  Preparedness.  Figure  6  illustrates  the  significant  relationship  between  the 
level  of  Leader  Involvement  during  training  and  Task  Perfonnance.  According  to  LSD  post-hoc 
tests,  Soldiers  in  units  with  substantial  levels  of  Leader  Involvement  (Macy  =  4.32,  SE  =  .1 1)  were 
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more  likely  to  report  higher  task  performance  compared  to  Soldiers  in  those  units  with  only  some 
level  of  Leader  Involvement  (Ma  dj  =  3.95,  SE  =  .06  ,p  =  .01)  and  units  with  no  Leader 
(dLadj  =  3.83,  SE  =  .13,  p  =  .01).  The  difference  in  reported  levels  of  Task 
is  not  statistically  significant  for  those  Soldiers  in  units  with  some  level  of  Leader 
compared  to  those  Soldiers  in  units  with  no  Leader  Involvement  during  training. 
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No  Leader  Some  Leader  Substantial  Leader 

Involvement  Involvement  Involvement 


Level  of  Leader  Involvement  during  Training 


*  Adjusted  means  significantly  different  from  one  another,  F  (2,  136)  =  3.67,  rf  =  .05,  p  <  .05 


Figure  6.  Impact  of  leader  involvement  on  task  perfonnance. 


Involvement 

Performance 

Involvement 


Figure  6  depicts  the  relationship  between  the  level  of  Leader  Involvement  during  training 
and  Training  Motivation.  As  can  be  seen  in  this  figure,  the  highest  Training  Motivation  levels 
were  reported  by  those  trainees  with  substantial  levels  of  Leader  Involvement  (Madj  =  4.24,  SE  = 
.14).  LSD  post-hoc  tests  demonstrated  that  both  trainees  with  some  (Madj  =  3.82,  SE  =  .08,/?  < 
.05)  and  no  (Madj  =  3.41,  SE  =  .11, p  <  .01)  Leader  Involvement  reported  significantly  lower 
levels  of  motivation.  The  difference  in  motivation  levels  for  Soldiers  in  units  with  no  Leader 
Involvement  and  some  Leader  Involvement  is  also  statistically  significant  (p  <  .05). 

In  general,  these  results  demonstrate  that  GBT  can  have  the  power  to  impact  individual- 
level  learning,  especially  in  an  appropriate  context.  Being  prepared  for  the  training  seems  to  be 
one  key  element  for  individuals  to  learn  from  GBT.  Also,  having  unit  leadership  involved  during 
the  training  seems  to  provide  support  for  individuals  to  be  more  motivated  and  perform  better 
during  the  training.  The  next  section  examines  similar  relationships  with  unit-level  variables. 
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No  Leader  Some  Leader  Substantial  Leader 

Involvement  Involvement  Involvement 


Level  of  Leader  Involvement  During  Training 


*  Adjusted  means  significantly  different  from  one  another,  F (2,  138)  =  5.41,  rf  =  .01, p  <  .01 
Figure  7.  Impact  of  leader  involvement  on  training  motivation. 


Unit  Level  Analyses 

Hypotheses  4  through  6  focused  on  the  influence  of  the  training  and  the  training  context 
on  unit  level  variables  (Unit  Process,  Unit  Cohesion,  Unit  Efficacy,  and  Unit  Effectiveness).  The 
first  analysis  with  the  unit  level  variables  was  to  examine  the  general  impact  of  the  training. 
Paired  sample  t-tests  were  conducted  to  examine  whether  trainees  perceived  higher  levels  of 
these  unit  level  variables  following  the  training  as  opposed  to  prior  to  the  training.  Although  all 
of  the  post-test  scores  were  higher  than  the  pre-test  scores,  the  only  significant  differences 
existed  with  Unit  Process  and  Unit  Cohesion.  Table  2  showcases  the  results  of  this  analysis  for 
all  four  unit  level  outcomes.  These  results  indicate  that  Soldiers  reported  feeling  better  about 
how  their  unit  engages  in  various  team  processes  (e.g.,  communication,  backing  each  other  up) 
and  the  level  of  cohesion  present  among  unit  members  after  having  participated  in  training  with 
VBS2:  U.S.  Army. 
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Table  2 


T-Test  Results  Comparing  Pre-  to  Post-Training  Outcome  Variables 


Comparison 

M 

SD 

t 

Unit  Process 

Pre  Training 

3.30 

.70 

-2.71* 

Post  Training 

3.46 

.78 

Unit  Cohesion 

Pre  Training 

3.41 

.68 

-1.93* 

Post  Training 

3.54 

.89 

Unit  Efficacy 

Pre  Training 

3.46 

.78 

-1.33 

Post  Training 

3.55 

.82 

Unit  Effectiveness 

Pre  Training 

3.07 

.78 

-1.40 

Post  Training 

3.16 

.83 

*  P  fL-05 


Hypotheses  5  and  6  focused  on  the  impact  of  situational  factors  on  training  effectiveness 
at  the  unit  level.  As  with  the  individual  level  variables,  the  effect  of  Unit  Preparation  and  Leader 
Involvement  on  the  unit  level  outcomes  was  examined  through  ANCOVA,  controlling  for  the 
impact  of  the  platoon  that  the  trainees  were  in  as  well  as  pre-training  levels  of  the  variables. 
Results  showed  that  the  level  of  Unit  Preparation  significantly  impacted  a)  Unit  Process,  b)  Unit 
Cohesion,  d)  Unit  Efficacy,  and  d)  Unit  Effectiveness.  Figures  7  through  10  depict  the  nature  of 
these  relationships.  As  seen  in  the  four  graphs,  higher  levels  of  unit  level  variables  were  reported 
by  Soldiers  in  units  that  had  prepared  more  for  the  training  exercises.  These  analyses  indicate 
that  the  VBS2:  U.S.  Anny  training  was  particularly  effective  at  increasing  how  unit  members 
work  together  and  feel  about  working  together  when  units  have  adequately  prepared  for  that 
training. 
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Less  Prepared  More  Prepared 

Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  126)  =  10.23,  rf  =  .08,  p  <  .01 
Figure  8.  Impact  of  unit  preparation  on  unit  process. 


Less  Prepared  More  Prepared 

Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  126)  =  6.72,  rf  =  .05,  p  <  .05 
Figure  9.  Impact  of  unit  preparation  on  unit  cohesion. 
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Less  Prepared  More  Prepared 

Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  125)  =  3.95,  rf  =  .03,  p  <  .05 
Figure  10.  Impact  of  unit  preparation  on  unit  efficacy. 


Less  Prepared  More  Prepared 

Unit  Preparation 

*  Adjusted  means  significantly  different  from  one  another,  F  (1,  125)  =  5.00,  rf  =  .04,  p  <  .05 
Figure  1 1 .  Impact  of  unit  preparation  on  unit  effectiveness. 

Finally,  Hypothesis  6  predicted  that  Leader  Involvement  would  positively  impact  the  four 
unit  level  variables.  Support  was  found  for  only  one  part  of  that  hypothesis;  specifically,  the 
level  of  Leader  Involvement  was  significantly  related  to  Unit  Process.  This  relationship  is 
illustrated  in  Figure  1 1.  As  can  be  seen  in  this  graph,  Soldiers  in  units  with  substantial  levels  of 
leader  involvment  perceived  the  highest  levels  of  Unit  Process  (Ma  dj  =  3.90,  SE  =  .  17).  Post-hoc 
LSD  tests  demonstrate  that  this  reported  level  of  Unit  Process  is  statistically  different  compared 
to  Soldiers  in  units  with  some  Leader  Involvement  (Madj  =  3.30,  SE  =  .09,  p  <  .05)  and  no  Leader 
Involvement  (Ma ^  =  3.19,  SE  =  .20,  p  <  01).  The  difference  in  Unit  Process  levels  perceived  by 
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Soldiers  in  units  with  no  Leader  Involvement  and  some  Leader  Involvement  was  not  statistically 
significant. 


No  Leader  Some  Leader  Substantial  Leader 

Involvement  Involvement  Involvement 

Level  of  Leader  Involvement  during  Training 


*  Adjusted  means  significantly  different  from  one  another,  F  (2,  137)  =  4.02,  rf  =  .06 ,p<  .05 
Figure  12.  Impact  of  leader  involvement  on  unit  process. 


Discussion  and  Conclusions 

This  report  provides  some  evidence  that  training  using  VBS2:  U.S.  Army  is  effective  in 
improving  both  individual-level  and  unit-learning  learning.  In  terms  of  individual-level  learning, 
the  training  did  not  directly  impact  how  prepared  Soldiers  felt  to  perform  certain  skills;  however, 
this  non-significant  finding  seems  to  be  more  of  a  function  of  measurement  than  the  actual 
training.  The  lack  of  consistency  across  the  training  scenarios,  and  consequentially  the  training 
objectives,  makes  it  difficult  to  measure  the  appropriate  skills.  That  is,  our  list  of  skills,  prepared 
in  advance  of  the  training  and  without  knowledge  of  the  unit’s  training  objectives,  may  simply 
not  have  matched  well  with  the  skill(s)  the  unit  wished  to  train.  However,  after  engaging  in  the 
training,  Soldiers  reported  feeling  more  prepared  to  plan  a  convoy  operation  than  prior  to 
training.  Even  though  specific  training  objectives  differed  from  unit  to  unit,  one  commonality 
across  all  of  the  training  sessions  was  that  each  mission  focused  on  planning  for  and  conducting 
convoy  operations.  Therefore,  it  makes  sense  that  Soldiers  would  report  feeling  more  prepared  to 
do  this  type  of  planning  after  actually  having  done  it  through  training  using  VBS2:  U.S.  Army. 
This  finding  has  implications  for  evaluating  GBT,  as  well  as  presents  evidence  for  the  validity  of 
GBT.  First,  it  points  to  the  importance  of  targeting  and  measuring  clear  learning  objectives  when 
using  GBT.  Second,  this  finding  illustrates  that  when  clear  learning  objectives  are  present  in 
games,  games  can  effectively  prepare  trainees  to  some  extent  and  make  them  feel  more  confident 
in  their  ability  to  perform  skills  targeted  during  the  training. 

Results  associated  with  understanding  the  impact  of  GBT  on  individual  level  learning 
also  point  to  the  importance  of  the  context  surrounding  the  training.  Particularly,  the  extent  to 
which  units  prepared  for  the  training  and  how  involved  unit  leadership  was  during  the  training 
influence  training  effectiveness.  Although  both  of  these  contextual  variables  positively  impacted 
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the  individual  level  variables,  unanswered  questions  still  remain  about  their  impact.  Future 
research  needs  to  further  examine  the  nature  of  Leader  Involvement  in  GBT  to  understand 
exactly  how  it  contributes  to  effectiveness.  For  example,  is  it  enough  for  leaders  to  just  be 
present  during  training,  or  do  they  need  to  provide  a  certain  type  of  feedback  to  be  influential? 
Additional  research  should  focus  on  uncovering  this  relationship  in  more  depth. 

Results  of  this  analysis  suggest  that  the  GBT  investigated  here  is  particularly  useful  for 
increasing  unit-level  outcomes.  In  general,  the  training  influenced  how  the  unit  felt  they  worked 
together  (Unit  Process),  unit  motivational  states  (Unit  Cohesion  and  Unit  Efficacy),  and  the 
perceived  effectiveness  and  performance  of  the  unit.  Trainees  reported  more  effective  unit 
processes  and  increased  levels  of  unit  cohesion  from  pre-  to  post-training.  In  addition,  Soldiers  in 
units  that  were  more  prepared  for  the  training  reported  higher  levels  of  Unit  Process,  Unit 
Cohesion,  Unit  Efficacy,  and  Unit  Effectiveness  compared  to  Soldiers  in  units  with  lower  levels 
of  preparation.  It  is  likely  that  units  who  spent  time  preparing  for  the  training  exercises 
emphasized  and  focused  on  getting  the  unit  to  work  together.  Anecdotal  evidence  from  our 
training  observations  and  data  collections  indicates  that  unit  leaders  were  using  the  training  as  a 
chance  to  focus  on  unit  level  behaviors.  They  continually  emphasized  the  importance  of 
communicating  with  each  other  during  the  training  and  understanding  the  role  and  position  of 
each  unit  member.  The  results  with  the  unit  level  variables  mirror  this  emphasis  by  the  units 
prior  to  and  during  the  training. 

In  contrast  to  the  influence  of  Unit  Preparation  on  all  four  unit-level  variables,  the  level 
of  Leader  Involvement  in  the  training  only  influenced  the  level  of  Unit  Process  reported  by  the 
trainees.  As  pointed  out  with  the  individual-level  variables,  understanding  the  nature  of  this 
involvement  may  help  to  explain  why  process  was  the  only  variable  influenced.  When  unit 
leadership  was  involved  during  the  training,  their  involvement  likely  came  in  the  form  of 
feedback  about  how  the  unit  was  working  together  (i.e.,  their  team  processes)  and  was  not 
centered  on  motivational  states  or  overall  performance  feedback. 

Analysis  Limitations 

There  are  several  limitations  associated  with  this  analysis,  many  of  them  associated  with 
the  challenges  in  collecting  data.  As  indicated  earlier  in  this  report,  there  was  a  very  limited 
amount  of  time  available  to  collect  data.  Therefore,  the  measures  administered  had  to  be  limited 
and  mainly  self-report  in  nature.  Second,  there  was  no  opportunity  for  a  control  group.  Although 
we  were  able  to  gather  both  pre-  and  post-training  data,  the  results  of  this  analysis  would  have 
been  stronger  if  we  had  been  able  to  compare  the  GBT  to  a  group  that  did  not  receive  GBT.  In 
addition,  collecting  data  from  the  same  training  units  following  a  field  training  exercise  would 
provide  a  useful  analysis  of  transfer  of  training.  The  performance  of  units  which  engaged  in  GBT 
could  be  compared  to  the  performance  of  units  which  did  not  engage  in  GBT  to  more  deeply 
understand  how  effective  the  training  was.  Finally,  it  was  difficult  to  measure  objective  training 
performance  given  the  differences  in  training  objectives  across  the  units.  Each  of  these 
limitations  made  it  more  difficult  for  us  to  identify  significant  changes  or  improvements  as  a 
result  of  training  with  VBS2:  U.S.  Army.  Despite  these  limitations,  several  interesting  findings 
emerged  to  lend  credence  to  the  use  of  games  for  training  purposes.  Under  the  correct  conditions, 
GBT  can  be  an  engaging,  useful  training  mechanism. 
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Appendix  A-l 
Evaluation  Measures 

PARTICIPANT  INFORMED  CONSENT  FORM 

Date:  1  October  09 

Title:  Game-Based  Training  Effectiveness  (Unit) 

Purpose:  This  research  is  being  conducted  by  the  Army  Research  Institute  (ARI).  The  purpose  is  to 
obtain  information  about  the  effectiveness  of  computer  simulations  for  Army  training.  This  information 
will  be  used  to  help  determine  the  future  use  of  similar  simulations  in  the  Army. 

What  you  will  be  asked  to  do  in  this  study:  In  addition  to  normal  training  activities,  you  will  be  asked 
to  complete  a  short  background  questionnaire,  several  paper  and  pencil  tests,  and  several  questionnaires 
about  your  training. 

Time  required:  It  is  estimated  that  all  questionnaires  will  take  no  longer  than  30  minutes  to  complete  in 
total. 

Cost/Risk:  There  are  no  associated  risks  with  participating  in  this  research.  Providing  information  is 
voluntary,  and  you  have  the  right  to  withdraw  from  the  research  at  any  time  without  consequences. 

Failure  to  respond  to  any  particular  question  will  not  result  in  any  penalty.  All  of  your  responses  will  be 
kept  confidential  and  will  not  be  shared  with  other  individuals  participating  in  this  research.  Any 
identifying  information  collected  will  be  used  only  by  persons  engaged  in,  and  for  the  purposes  of,  the 
research. 

Benefits:  The  benefits  that  you  may  expect  from  your  participation  in  this  research  are  minimal.  Your 
participation  will  aid  efforts  to  improve  the  effectiveness  of  Army  training. 

Voluntary  consent:  Your  signature  below  means  that  you  have  read  this  consent  form  and  agree  to 
voluntarily  participate  in  this  study.  Y our  signed  consent  form  will  be  kept  separate  from  the  rest  of  the 
research  materials.  Thus,  your  name  will  not  be  associated  with  your  responses,  and  the  data  that  you 
provide  will  be  anonymous.  You  must  be  18-years  of  age  or  older  to  participate. 

•  I  have  read  the  procedure  described  above. 

•  I  have  been  informed  of  the  reasons  for  this  research. 

•  I  have  had  the  research  explained  to  me. 

•  I  have  had  all  of  my  questions  answered. 

•  I  give  consent  voluntarily  to  participate  in  this  research. 

•  By  signing  this  consent  form,  I  have  not  given  up  any  of  my  legal  rights. 


Signature  of  Participant 


Date 


Printed  Name 


Signature  of  Person  Conducting  Informed  Consent  Discussion  Date 
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Privacy  Act  Statement 

Title:  Game-Based  Training  Effectiveness  (Unit) 

Date:  1  October  09 

AUTHORITY:  The  Department  of  the  Army  may  collect  the  information  requested  in  this  session  under 
the  authority  of  10  United  States  Code,  Section  2358,  “Research  and  Development  Projects.”  In 
accordance  with  the  Privacy  Act  of  1974  (Public  Law  93-579),  this  notice  informs  you  of  the  purpose, 
use,  and  confidentiality  of  this  session. 

PURPOSE:  The  purpose  is  to  collect  obtain  information  about  the  effectiveness  of  computer  simulations 
for  Army  training. 

ROUTINE  Uses:  This  information  will  be  used  to  help  determine  the  future  use  of  similar  simulations  in 
the  Army. 

DISCLOSURE:  Participating  in  this  session  is  voluntary  and  you  may  choose  at  any  time  not  to 
participate.  There  is  no  penalty  for  choosing  not  to  participate. 

Confidentiality:  We  will  not  identify  you,  or  attribute  responses  to  any  particular  participant 
made  during  this  session,  and  we  will  NOT  include  your  name  or  other  personally  identifiable 
information  in  our  interview  notes  or  report.  Likewise,  we  ask  that  you  respect  the  confidential 
nature  of  this  session,  by  not  identifying  individual  participants  with  comments  made  or  heard 
during  this  session. 

We  cannot  provide  "confidentiality"  or  "non-attribution,"  to  a  participant  regarding  comments  involving 
criminal  activity/behavior,  or  statements  that  pose  a  threat  to  yourself  or  others.  Do  NOT  discuss  or 
comment  on  classified  or  operationally  sensitive  information  during  this  session. 

CONTACT:  For  further  information  about  this  project  or  your  rights  as  a  participant,  send  e-mail  to: 
ARI_RES@conus.army.mil  (type  “GBT  Unit”  in  the  subject  line). 


RETAIN  FOR  YOUR  RECORDS 


24 


Demographics 


1.  Age 

2.  Civilian  Education  (check  highest  level  attained): 

□  GED  □  High  □  Some  □  B.S.  /B.A.  □ 

Post¬ 

School  College 

college 

3.  Time  in 

4.  Grade/ 

5.  MOS/ 

Service: 

Yrs  Mos.  Rank: 

Branch 

6.  Deployment  Experience  (Provide  information  for  all  that  apply) 


Position(s) 

Time  (months) 

OIF 

OEF 

Bosnia 

7.  Have  you  participated  in  VBS2  training  before  today?  □  Yes  □  No 

If  yes,  how  many  times  prior  to  today  have  you  participated  in  VBS2  training?  _ 

8.  Using  the  scale  below,  indicate  how  familiar  are  you  with  the  other  Soldiers  that  you  are  training  with 

today;  please  think  about  familiarity  in  terms  of  training  together,  deployed  together,  and  even  just 
knowing  each  other  in  a  more  social  aspect:  _ 

1  -  Completely  unfamiliar  (e.g.,  I  have  not  met  most  of  them  prior  to  today).  2  -  Slightly  unfamiliar.  3  - 
Neutral.  4  -  Slightly  familiar.  5  -  Very  familiar  (e.g.,  I  know  most  of  them  very  well) 
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Skill  Preparedness 


Prior  to  training  with  VBS2/Now  that  you  have  trained  with  VBS2,  please  respond  to  the  following  items  about 
how  prepared  you  feel  to  conduct  the  following  tasks;  please  use  the  following  scale  in  your  assessment  of 
each  task: 

1  -  Unprepared ,  2  -  Slightly  Unprepared,  3  -  Neither  Unprepared  nor  Prepared,  4  -  Slightly  Prepared,  5 
I've fiarrd 

_ 1 .  React  to  Attack  (Near  or  Far). 

_ 2.  Assess  the  tactical  situation. 

_ 3.  Consolidate  and  Reorganize. 

_ 4.  Locate  known  or  suspected  enemy  positions. 

_ 5.  Scan  my  sector  appropriately. 

_ 6.  Comply  with  rules  of  engagement. 

_ 7.  Control  convoy  movement  while  NOT  in  contact  with  the  enemy. 

_ 8.  Plan  a  tactical  convoy  operation. 

_ 9.  Break  contact. 

_ 10.  Coordinate  activities  with  your  chain  of  command. 

_ 1 1 .  Communicate  with  members  of  your  unit. 

_ 12.  Conduct  CASEVAC/Recovery  Operations. 

13.  React  to  IED 
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Training  Motivation 


Use  the  following  scale  for  the  following  items: 

1  -  Strongly  disagree,  2-  Disagree,  3  -  Neither  Agree  nor  Disagree,  4  -  Agree,  and  5  -  Strongly 
Agree 

1 .  I  enjoyed  this  training  exercise  very  much. 

2.  I  put  a  lot  of  effort  into  this  training  exercise. 

3.  It  was  important  to  me  to  do  well  in  this  training  exercise. 

4.  I  tried  very  hard  during  this  exercise. 

5.  This  training  exercise  was  fun. 

6.  I  would  describe  this  training  exercise  as  very  interesting. 

7.  I  am  satisfied  with  my  performance  in  this  training  exercise. 

8.  I  did  not  try  very  hard  during  this  exercise. 

9.  During  this  training  exercise,  I  was  thinking  about  how  much  I  enjoyed  it. 

10.  This  training  exercise  did  not  hold  my  attention. 
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Task  Performance 


Please  rate  your  ability  to  perform  each  of  the  following  tasks  in  the  simulator  at  the  end  of  today’s 

training.  Please  respond  using  the  following  scale: 

1  -Did  Not  Attempt,  2  -  Very  Poor,  3  -  Poor,  4  -  Good,  5  -  Very  Good 


Did  Not 

Very 

Very 

Attempt 

Poor 

Poor 

Good 

Good 

1 .  Move  tactically  while  dismounted. 

© 

© 

© 

© 

2.  Engage  a  target  with  my  individual  weapon. 

© 

© 

© 

© 

3.  Enter  a  building. 

© 

© 

© 

© 

© 

4.  Drive  a  vehicle. 

© 

© 

© 

© 

© 

5.  Change  my  posture  (prone-kneeling-standing) 

© 

© 

© 

© 

© 

6.  Detect  an  IED. 

© 

© 

© 

© 

© 

7.  Scan  for  threats. 

© 

© 

© 

© 

© 

8.  Identify  civilians. 

© 

© 

© 

© 

© 

9.  Communicate  with  others. 

© 

© 

© 

© 

© 

10.  Locate  the  source  of  enemy  fire. 

© 

© 

© 

© 

© 

1 1 .  Mount  a  vehicle. 

© 

© 

© 

© 

© 

12.  Distinguish  between  friendly  and  enemy  fire. 

© 

© 

© 

© 

© 

13.  Engage  a  target  with  a  vehicle-mounted  weapon. 

© 

© 

© 

© 

© 

14.  Identify  insurgents. 

© 

© 

© 

© 

© 

15.  Dismount  a  vehicle. 

© 

© 

© 

© 

© 
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Unit  Process 


Use  the  following  scale  for  the  following  items: 

1  -  Strongly  disagree,  2-  Disagree,  3  -  Neither  Agree  nor  Disagree,  4  -  Agree,  and  5  -  Strongly 
Agree 

1 .  My  training  unit  members  and  I  understand  how  one  another  prefer  to  communicate 
information  to  other  members. 

2.  My  training  unit  members  and  I  understand  when  we  need  to  consult  each  other. 

3.  My  training  unit  members  and  I  have  a  good  understanding  of  the  roles  played  by  each 
member. 

4.  My  training  unit  members  and  I  sometimes  feel  confused  about  how  members  should 
work  together. 

5.  My  training  unit  members  and  I  understand  how  members  of  the  unit  should  help  each 
other  out. 

6.  My  training  unit  members  and  I  understand  how  members  are  going  to  work  together  to 
achieve  our  goals. 

7.  My  training  unit  members  and  I  are  clear  about  what  tasks  each  member  is  responsible 
for. 

8.  My  training  unit  members  and  I  are  able  to  prevent  conflict  from  occurring  between 
members. 

9.  My  training  unit  members  and  I  are  able  to  manage  the  emotions  of  the  team  (e.g. 
frustration,  excitement). 

10.  My  training  unit  members  and  I  are  able  to  keep  confidence  high  within  the  unit. 

1 1 .  My  training  unit  members  and  I  are  able  to  keep  motivation  high  within  the  unit. 

12.  When  something  about  the  situation  is  uncertain,  more  knowledgeable  members  of  the 
training  unit  will  help  the  other  members  make  sense  of  the  situation. 

13.  When  one  person  is  struggling  with  a  task,  another  member  of  the  training  unit  will  step 
in  to  help. 

14.  My  training  unit  is  engaged  in  or  “into”  our  mission. 


29 


Unit  Cohesion 


Use  the  following  scale  for  the  following  items: 

1  -  Strongly  disagree,  2-  Disagree,  3  -  Neither  Agree  nor  Disagree,  4  -  Agree,  and  5  -  Strongly 
Agree 

1 .  My  training  unit  is  engaged  in  or  “into”  our  mission. 

2.  My  training  unit  members  enjoy  working  on  the  mission/tasks  we  must  collaborate  on. 

3.  It  is  important  for  my  training  unit  to  produce  high  quality  goals. 

4.  It  is  important  for  my  training  unit  members  to  perfonn  well  when  given  a  mission  and/or 
task. 

5.  The  members  of  my  training  unit  consider  the  mission/tasks  we  must  work  on  together  as 
meaningful  and  important. 

6.  My  training  unit  expects  to  benefit  from  successful  mission/task  perfonnance. 

7.  The  members  of  my  training  unit  like  each  other. 

8.  The  members  of  my  training  unit  get  along  with  one  another. 

9.  The  members  of  my  training  unit  feel  similar  to  one  another. 

10.  It  is  important  that  the  members  of  my  training  unit  socialize  while  working  together. 

1 1 .  The  members  of  my  training  unit  like  being  a  part  of  this  unit. 
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Unit  Efficacy 


Use  the  following  scale  for  the  following  items: 

1  -  Strongly  disagree,  2-  Disagree,  3  -  Neither  Agree  nor  Disagree,  4  -  Agree,  and  5  -  Strongly 
Agree 

1 .  I  am  confident  that  the  members  of  my  training  unit  and  I  can  effectively  set  contingency 
plans. 

2.  I  am  confident  that  the  members  of  my  training  unit  and  I  can  develop  good  strategies. 

3.  I  am  confident  that  the  members  of  my  training  unit  and  I  can  understand  the  tasks  at 
hand. 

4.  I  am  confident  that  the  members  of  my  training  unit  and  I  can  coordinate  effectively  with 
all  members. 

5.  I  am  confident  that  the  members  of  my  training  unit  and  I  can  detennine  our  progress 
towards  goals. 

6.  I  am  confident  that  the  members  of  my  training  unit  and  I  can  help  each  other  out  if 
necessary. 

7.  I  am  confident  that  the  members  of  my  training  unit  and  I  can  manage  conflict  and 
frustration. 


Unit  Effectiveness 

1 .  My  training  unit  performs  well  at  most  missions/tasks. 

2.  This  training  unit  is  effective. 

3.  Relative  to  other  training  units  perfonning  similar  functions,  this  training  unit  performs: 

a)  Better 

b)  The  Same 

c)  Worse 
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Unit  Preparation  for  Training 


Please  rate  the  preparedness  of  the  training  unit  using  a  5-point  scale  (1  =  Very  easy;  2  = 
Easy;  3  =  Average;  4  =  Difficulty;  5  =  Very  difficult).  Please  circle  your  answer. 

You  will  see  anchors  below  to  help  you  make  your  assessment.  Each  rating  has  several  factors 
for  you  to  consider  in  your  assessment.  Please  note  that  not  every  factor  has  to  be  true  in  order 
for  you  to  make  your  rating.  In  addition,  it  is  likely  that  some  additional  factors  may  influence 
your  assessment  of  Unit  Preparedness.  Please  do  not  consider  the  factors  below  as  the  definitive 
criteria,  but  instead,  use  them  as  a  guide  to  help  you  make  your  rating. 


Unit  Preparedness  Rating 

Factors  to  Consider  in  Rating 

1  (Completely  Unprepared) 

• 

• 

• 

Arrived  at  the  training  center  with  no  prior  communication  with 
the  training  facilitators  about  training  objectives 

No  definitive  training  objectives 

Spent  no  time  planning  for  the  mission  once  in  the  training  center 

2  (Slightly  Unprepared) 

• 

• 

• 

Arrived  at  the  training  center  with  no  prior  communication  with 
the  training  facilitators 

Flad  some  slightly  defined  training  objectives 

Spent  no  more  than  1 0  minutes  planning  for  the  mission  once  in 
the  training  center 

3  (Neither  Prepared  nor 
Unprepared) 

• 

• 

Talked  with  training  facilitators  once  to  convey  training  objectives 
Spent  no  more  than  1  hour  in  the  training  center  on  mission 
planning 

4  (Slightly  Prepared) 

• 

• 

• 

Talked  with  training  facilitators  more  than  once  to  convey  training 
objectives 

Discussed  ideas  with  the  training  facilitators  about  how  training 
missions  may  be  modified  to  meet  objectives 

Spent  at  least  a  half  day  planning  for  the  mission  in  the  training 
center  prior  to  conducting  mission 

5  (Completely  Prepared) 

• 

• 

• 

• 

Met  with  training  facilitators  multiple  times  prior  to  the  training 
exercise 

Provided  real-world  missions  for  the  training  facilitators  to 
incoiporate  into  the  training 

Worked  with,  and  provided  suggestions  to,  the  training  facilitators 
on  how  best  to  accomplish  training  objectives 

Spent  one  whole  day  in  the  training  center  conducting  mission 
planning  prior  to  conducting  the  actual  mission 
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Time  Spent  on  Training 


Time  spent  on  VBS2  training: 

minutes 

Time  spent  on  mission  briefing: 

minutes 

Time  spent  on  actual  mission: 

minutes 

Exercise  Management  -  Leader  Involvement 

Please  answer  each  of  the  below  questions  according  to  your  observations  during  the  unit  training. 

1 .  What  was  the  highest  level  of  unit  leadership  present  during  the  training  exercise? 

2.  Who  conducted  the  mission  briefing  (e.g.,  the  platoon  sergeant,  the  company  commander,  a 

training  facilitator)?  _ 

Aside  from  the  person  named  in  the  line  above,  who  else  helped  with  the  briefing? 

3.  If  feedback  was  provided  to  the  trainees  during  the  exercise,  who  provided  it? 

4.  If  someone  from  the  unit  participated  in  the  training  exercise  by  serving  as  headquarters,  who  was 

it?  _ 

5.  If  someone  from  the  unit  communicated  with  the  training  facilitators  during  the  exercise  to 

contribute  to  the  events  that  occurred  during  the  mission,  who  was  it?  _ 

6.  Who  conducted  the  AAR? _ 

Aside  from  the  person  named  in  the  line  above,  who  else  helped  with  the  AAR? 
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Appendix  B-l 

Skill  Preparedness  Items  -  Means  and  Standard  Deviations 


Comparison 

M 

SD 

t 

React  to  an  attack 

Pre  Training 

3.96 

.91 

1.09 

Post  Training 

3.85 

1.04 

Assess  the  tactical  situation 

Pre  Training 

3.89 

.90 

-.47 

Post  Training 

3.94 

.94 

Consolidate  and  reorganize 

Pre  Training 

3.84 

.94 

.14 

Post  Training 

3.83 

.99 

Locate  known  or  suspected  enemy  positions 

Pre  Training 

3.89 

.90 

.83 

Post  Training 

3.82 

.99 

Scan  my  sector  appropriately 

Pre  Training 

4.39 

.83 

.51 

Post  Training 

4.35 

.92 

Comply  with  rules  of  engagement 

Pre  Training 

4.12 

1.06 

-1.69 

Post  Training 

4.29 

.91 

Control  convoy  movement  while  NOT  in 
contact  with  the  enemy 

Pre  Training 

4.08 

.93 

-.64 

Post  Training 

4.14 

.96 

Plan  a  tactical  convoy  operation 

Pre  Training 

3.50 

1.15 

-2.61* 

Post  Training 

3.78 

1.06 

Break  contact 

Pre  Training 

3.77 

1.05 

-.54 

Post  Training 

3.82 

.99 

Coordinate  activities  with  your  chain  of 
command 

Pre  Training 

3.84 

.96 

-.64 

Post  Training 

3.90 

.98 

Communicate  with  members  of  your  unit 

Pre  Training 

4.20 

.82 

.68 

Post  Training 

4.13 

.92 

Conduct  CASEVAC/Recovery  operations 

Pre  Training 

3.68 

1.05 

-.66 

Post  Training 

3.75 

1.02 

React  to  IED 

Pre  Training 

3.94 

1.02 

-.98 

Post  Training 

4.04 

.99 

*  p  <  .01 
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